[ENH] Update digest to support latest Nipoppy processing status files #166

alyssadai · 2024-12-13T18:12:41Z

Changes proposed in this pull request:

Rename READMEs, schemas, and example input files to remove "bagel" to avoid confusion with files used/generated by Neurobagel / Nipoppy (input files are now just called digest files or inputs)
- Note: this was done for all user-facing files, but the code internally still uses "bagel" in a lot of places to refer to the input data. These will be updated in Rename functions and variables to deprecate "bagel" and reflect non-imaging data #164
Update imaging digest schema to align with latest Nipoppy processing status file schema
- Rename identifier columns to match Nipoppy
- Remove columns no longer needed: PHASE__, STAGE__, HAS_DATATYPE__, HAS_IMAGE, has_mri_data
- Make pipeline_starttime optional
- Add columns: pipeline_step, status
- Remove no-longer-needed IsPrefixedColumn, MissingValue column properties
- The digest still has a few additional columns for now that may be useful in processing status files in future
- See also Consider removing column categories in digest schemas #163
Use combination of pipeline-version-step instead of pipeline-version for tracking completion
- Each pipeline-version-step combination now has distinct plots
Accept TSV instead of CSV inputs (Nipoppy now only produces TSVs)
Regenerate reference example inputs as TSVs, with column names updated according to latest schema
- Reference example imaging digest file now based on https://github.com/neurobagel/neurobagel_examples/blob/main/data-upload/nipoppy_proc_status_synthetic.tsv
Regenerate test example files as TSVs, with column names updated according to latest schema
Update tests (mostly renaming column references)
Update pre-generated QPN file paths based on https://github.com/neurodatascience/qpn_workflows

For reviewer: you can test out the changes in the digest using either of these files:

Checklist

This section is for the PR reviewer

PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see our Contributing Guidelines for more info)
PR links to GitHub issue with mention Closes #XXXX
Tests pass
Checks pass

For new features:

Tests have been added

For bug fixes:

There is at least one test that would fail under the original bug conditions.

- match column names to https://nipoppy.readthedocs.io/en/latest/schemas/index.html#imaging-bagel-file - make pipeline_starttime optional - remove "PHASE__" & "STAGE__" cols, "step" & "status" cols - simplify column descriptions

- "HAS_DATATYPE__", "HAS_IMAGE__"

- remove "IsPrefixedColumn" - rename ID columns & update their descriptions

- refactor out primary session column name into a variable

alyssadai · 2024-12-13T19:18:40Z

Hey @michellewang, requesting your review here on just the digest schema changes as well as text including READMEs and status descriptions. Will tag you on the relevant files/sections under files changed!

README.md

schemas/README.md

schemas/imaging_digest_schema.json

schemas/phenotypic_digest_schema.json

digest/utility.py

michellewang

@alyssadai let me know if I missed anything!

README.md

digest/utility.py

schemas/README.md

schemas/imaging_digest_schema.json

schemas/phenotypic_digest_schema.json

surchs

Thanks @alyssadai. Changes look good, I did flag one thing I believe is an accidental change/bug with pd.read_csv to pd.read_tsv. I haven't tried this locally, but I'm pretty sure that doesn't work. Even if I'm wrong about that, I'd say add a test that covers the load_file_from_path route in utility, because it doesn't seem tested yet.

digest/utility.py

digest/app.py

Co-authored-by: Michelle Wang <tomichellewang@gmail.com>

Co-authored-by: Sebastian Urchs <surchs@users.noreply.github.com>

…neurobagel/dash into update-proc-status-file-parsing

…hema

alyssadai · 2025-01-13T18:20:56Z

Blocked from merging until https://github.com/neurodatascience/qpn_workflows is updated with latest QPN digest files.

surchs

Great stuff @alyssadai 🎉

🧑‍🍳

schemas/imaging_digest_schema.json

alyssadai added 26 commits December 9, 2024 01:24

update proc status file schema

3802068

- match column names to https://nipoppy.readthedocs.io/en/latest/schemas/index.html#imaging-bagel-file - make pipeline_starttime optional - remove "PHASE__" & "STAGE__" cols, "step" & "status" cols - simplify column descriptions

remove columns about raw imaging data and IsPrefixedColumn property

38bff3a

- "HAS_DATATYPE__", "HAS_IMAGE__"

require TSV inputs instead of CSVs

4dd3d1f

align phenotypic file schema with proc status file changes

1b14759

- remove "IsPrefixedColumn" - rename ID columns & update their descriptions

update id column references and handling to reflect new names

669adda

- refactor out primary session column name into a variable

update comment

5880361

fix imports

84dda4a

update references to renamed 'pipeline-complete' column

403ed1f

remove MissingValue column property from schema

e7df57d

refactor id column extraction

53047f0

update docstrings

4745d7f

update README

07c11d3

update README

e53c76d

update README

3e10487

update schema README

99cfdf5

update schema README

b25d14e

rename schemas

902947a

fix sneaky outdated session column reference

1f72a13

update columns in test data digests and convert to TSVs

b2dde0e

update missing column example based on revised schema

037e5dc

rename test data files

745483a

update docstring

a18a2d6

rename reference example inputs and update symlinks

cab23fb

update tests

3d492a6

replace csv with tsv in docstrings

e622585

rename PRIMARY_SESSION var

08a5812

alyssadai mentioned this pull request Dec 13, 2024

Update the digest to work with the latest imaging processing status file #158

Closed

alyssadai marked this pull request as ready for review December 13, 2024 19:16

alyssadai requested a review from michellewang December 13, 2024 19:17